AequeVox: Automated Fairness Testing of Speech Recognition Systems

Abstract Automatic Speech Recognition (ASR) systems have become ubiquitous. They can be found in a variety of form factors and are increasingly important our daily lives. As such, ensuring that these equitable to different subgroups the population is crucial. In this paper, we introduce, AequeVox , an automated testing framework for evaluating fairness ASR systems. simulates environments assess effectiveness populations. addition, investigate whether chosen simulations comprehensible humans. We further propose fault localization technique capable identifying words not robust varying environments. Both components able operate absence ground truth data. evaluate on speech from four datasets using three commercial ASRs. Our experiments reveal non-native English, female Nigerian English speakers generate 109% 528.5% 156.9% more errors, average than native male UK Midlands speakers, respectively. user study also reveals 82.9% (employed through transformations) had comprehensibility rating above seven (out ten), with lowest being 6.78. This validates violations discovered by . Finally, show non-robust words, as predicted embodied 223.8% errors across all